Skip to content

perf(recording): Windows fragmented M4S readback overlap and GPU scale - #3586

Open
AatmanAJ wants to merge 1 commit into
CapSoftware:mainfrom
AatmanAJ:perf/win-fragmented-gpu-encode
Open

AatmanAJ wants to merge 1 commit into
CapSoftware:mainfrom
AatmanAJ:perf/win-fragmented-gpu-encode

Conversation

@AatmanAJ

@AatmanAJ AatmanAJ commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Fragmented M4S (instant + studio default on Windows): Split D3D→FFmpeg readback from H.264 encode on a bounded staging channel so capture can keep producing while the prior frame encodes. When the encoder would have used internal swscale (scale and/or BGRA→YUV), enable prefer_external_conversion and feed frames through AsyncConverterPool (D3D11 video processor when available) with queue_preconverted_frame_to, overlapping GPU/CPU conversion with readback and libx264/NVENC work.
  • Instant mode capture sizing: Apply the same max-width capture constraint on Windows as macOS so VideoInfo matches the encode resolution instead of capturing native 4K and scaling twice (capture scaler + encoder swscale).
  • Display resize scaling: Prefer D3D11 VideoProcessorBlt for dimension mismatches and read back only at the target resolution; fall back to the existing CPU swscale path when the GPU scaler cannot be created.

Not in this PR (follow-up)

  • Wiring Media Foundation H.264 from WindowsMuxer into fragmented M4S (zero-copy texture encode for studio/instant segments).
  • Eliminating the final BGRA CPU readback entirely (would require MF or hardware FFmpeg with D3D surfaces).

Test plan

  • Windows instant recording at 1920p cap on a >1080p display: verify lower CPU vs main, stable frame pacing, acceptable quality.
  • Windows studio fragmented recording (display): same smoke test.
  • Resize display during recording: confirm GPU scale path logs and no corruption.
  • cargo check -p cap-recording (CI).

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge based on the reviewed code.

Summary

This PR overlaps Windows frame readback, conversion, and H.264 encoding.

  • Windows M4S recordings can read back frames while earlier frames are converted and encoded.
  • RGB-to-YUV conversion now uses the BT.709 color matrix.
  • Windows instant recordings capture at the configured output-size limit.
  • Windows display resizing scales frames on the GPU when the video processor is available.

No new actionable issues were established. All six previous, unnumbered findings are addressed in the current code: module placement, the unsafe call, frame ordering, colors, failed-setup caching, and sequence advancement.

Checks were limited to source review; no build or runtime tests were run.

Reviews (6) · Last reviewed commit: "perf(recording): overlap Windows fragmen..." · Reviewed by Greptile

@vercel

vercel Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Someone is attempting to deploy a commit to the Cap Software Inc Team on Vercel.

A member of the Team first needs to authorize it.

@AatmanAJ

Copy link
Copy Markdown
Contributor Author

@greptile-apps please re-review. Addressed the open P1/P2 items in the latest push on this branch.

@AatmanAJ
AatmanAJ force-pushed the perf/win-fragmented-gpu-encode branch from 4e93812 to e077503 Compare October 30, 2026 00:26
@AatmanAJ

Copy link
Copy Markdown
Contributor Author

@greptile-apps review

2 similar comments
@AatmanAJ

Copy link
Copy Markdown
Contributor Author

@greptile-apps review

@AatmanAJ

Copy link
Copy Markdown
Contributor Author

@greptile-apps review

@richiemcilroy

Copy link
Copy Markdown
Member

Thanks! Can you rebase on main to clear the conflicts and resolve the open review threads? Then we'll take a look.

…cale

Rebased onto main. The split readback/encode thread now reports encode
failures through report_encode_result and handles main's mid-recording
hardware-to-software encoder fallback: queue_preconverted_frame recovers
like queue_frame, and the converter pool is rebuilt when the fallback
encoder needs a different input format.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AatmanAJ
AatmanAJ force-pushed the perf/win-fragmented-gpu-encode branch from ea8bf7c to 9633795 Compare November 4, 2026 17:58
@AatmanAJ

AatmanAJ commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

@greptile-apps review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants